DOMAIN:¶
Semiconductor manufacturing process
CONTEXT:¶
A complex modern semiconductor manufacturing process is normally under constant surveillance via the monitoring of signals/variables collected from sensors and or process measurement points. However, not all of these signals are equally valuable in a specific monitoring system. The measured signals contain a combination of useful information, irrelevant information as well as noise. Engineers typically have a much larger number of signals than are actually required. If we consider each type of signal as a feature, then feature selection may be applied to identify the most relevant signals. The Process Engineers may then use these signals to determine key factors contributing to yield excursions downstream in the process. This will enable an increase in process throughput, decreased time to learning and reduce the per unit production costs. These signals can be used as features to predict the yield type. And by analysing and trying out different combinations of features, essential signals that are impacting the yield type can be identified.
DATA DESCRIPTION:¶
sensor-data.csv : (1567, 592) The data consists of 1567 datapoints each with 591 features. The dataset presented in this case represents a selection of such features where each example represents a single production entity with associated measured features and the labels represent a simple pass/fail yield for in house line testing. Target column “ –1” corresponds to a pass and “1” corresponds to a fail and the data time stamp is for that specific test point.
PROJECT OBJECTIVE:¶
We will build a classifier to predict the Pass/Fail yield of a particular process entity and analyse whether all the features are required to build the model or not.
1. Import and understand the data.¶
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer
import matplotlib.pyplot as plt
import seaborn as sns
from imblearn.over_sampling import SMOTE
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn import svm
from sklearn.metrics import accuracy_score,confusion_matrix,recall_score,classification_report,roc_auc_score,roc_curve
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import GradientBoostingClassifier
from xgboost import XGBClassifier
from sklearn.model_selection import StratifiedKFold
import pickle
#A. Import ‘signal-data.csv’ as DataFrame.
print("----Signal data----\n")
Signaldf=pd.read_csv('signal-data.csv')
print("Shape",Signaldf.shape)
Signaldf.head()
#B. Print 5 point summary and share at least 2 observations
Signaldf.describe(include='all')
Signaldf.info()
Observations
- variables are in different range , need to scale the data.
- some of the Features seems to be skewed and there are lot of outliers in the data.
- Data have high percentage of null values. Based on threshold limit some of the null features need to be removed and some of the features with negligible amount of null values should be imputed.
- Standard deviation of some feature are very high [Eg. feature 3 have SD of 441.69 ].
- There are feature with 0 standard deviation implies same constant for all the rows of feature [Eg. Feature=5]
2. Data cleansing:¶
#A. Write a for loop which will remove all the features with 20%+ Null values and impute rest with mean of the feature.
featurewithnull=0
feature_morethan20per_null=0
removedfeature=[]
for feature in Signaldf.columns:
#if data missing
if(Signaldf[feature].isnull().sum()):
featurewithnull=featurewithnull+1
missing_per=round((Signaldf[feature].isnull().sum()/len(Signaldf[feature]))*100,2)
#Missing percentage >= 20
if(missing_per>=20):
feature_morethan20per_null=feature_morethan20per_null+1
Signaldf=Signaldf.drop(feature, axis=1)
removedfeature.append(feature)
#Missing percentage < 20 imputing with mean
else:
Signaldf[feature].fillna( round(Signaldf[feature].mean(),4), inplace=True)
print("Number of feature which has null values",featurewithnull)
print("Number of feature which has more than 20 percentage null values",feature_morethan20per_null)
print("Shape after missing value treatment ",Signaldf.shape)
#B. Identify and drop the features which are having same value for all the rows.
count=0
for feature in Signaldf.columns[1:]:
if Signaldf[feature].std()==0:
Signaldf.drop([feature],axis=1,inplace=True)
removedfeature.append(feature)
count=count+1
print("No. of columns whose standard deviation was 0 and hence dropped:",count)
print("Shape after Missing value treatment and removing feature with same value for all rows",Signaldf.shape)
#C. Drop other features if required using relevant functional knowledge. Clearly justify the same.
#Time feature will not have predict value to the target
Signaldf.drop(['Time'],axis=1,inplace=True)
removedfeature.append('Time')
print ("Time feature is removed")
#Low variance filter
col = Signaldf.columns.values
from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=0.2)
sel.fit_transform(Signaldf)
features = col[(sel.get_support(indices=True))]
#print(sel.get_support())
#print(features)
Signaldf = Signaldf.filter(features)
#print(Signaldf.columns)
print("No. of columns with less variance and hence dropped:",len(col)-len(features))
Time feature will not have predict value to the target
Feature with less variance is likely to be useful for classifying the target. In the previous step already zero variance feature are removed that is feature with same constant for all the values in the row are removed.
Further removing the features with less than 0.2 variance and reducing the dimension
print("Shape after low variance filter",Signaldf.shape)
#D. Check for multi-collinearity in the data and take necessary action
corr_matrix = Signaldf.corr()
iters = range(len(corr_matrix.columns) - 1)
drop_cols = []
# Iterate through the correlation matrix and compare correlations
for i in iters:
for j in range(i+1):
item = corr_matrix.iloc[j:(j+1), (i+1):(i+2)]
col = item.columns
row = item.index
val = abs(item.values)
# If correlation exceeds the threshold
if val >= 0.70:
# Print the correlated features and the correlation value
print(col.values[0], "X", row.values[0], "=", round(val[0][0], 2))
drop_cols.append(col.values[0])
drops = set(drop_cols)
Signaldf = Signaldf.drop(columns=drops)
print("No. of columns who are highly correlated and hence dropped: ",len(drops))
print("Shape after multicolinearity feature removal",Signaldf.shape)
#E. Make all relevant modifications on the data using both functional/logical reasoning/assumption
#Analysing data
Signaldf.describe(include='all').T
Feature [419,499,500,511,521]- 50% of data are 0 Action 1:. removing those features
outliers are Observed in many features Action 2:impute the outliers with lower whisker and upper whisker accordingly
Target value are in -1 and 1 --Action 3: PASS/fail shuld be cnverted to 0,1 for easy interpretation
#Action 1
count=0
for feature in Signaldf.columns[:-1]:
if (np.percentile(Signaldf[feature],50)==0):
Signaldf.drop([feature],axis=1,inplace=True)
removedfeature.append(feature)
print(feature)
count=count+1
print("No. of columns who have 50% of data as 0 and hence dropped: ",count)
print("Shape after removing majority of zero feature",Signaldf.shape)
#Action 2
def outlierdetection(df):
print(" \n------ Outlier Detection ------\n")
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
#print(IQR)
outliers=np.where((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR)))
if (len(outliers[0])==0):
print("No Outliers Found")
else:
print(outliers)
outlierhandled=outliertreatment(df,Q1,Q3,IQR)
return outlierhandled
#Outlier treatment
def outliertreatment(df,Q1,Q3,IQR):
# Replace every outlier on the lower side by the lower whisker
for i, j in zip(np.where(df < Q1 - 1.5 * IQR)[0], np.where(df < Q1 - 1.5 * IQR)[1]):
whisker = Q1 - 1.5 * IQR
df.iloc[i,j] = whisker[j]
#Replace every outlier on the upper side by the upper whisker
for i, j in zip(np.where(df > Q3 + 1.5 * IQR)[0], np.where(df > Q3 + 1.5 * IQR)[1]):
whisker = Q3 + 1.5 * IQR
df.iloc[i,j] = whisker[j]
return df
Signaldf2=Signaldf.copy()
Signaldf2.iloc[:,:-1]=outlierdetection(Signaldf.iloc[:,:-1])
print("Shape after data pre-prcessing",Signaldf2.shape)
#target balance
Signaldf2['Pass/Fail'].value_counts()
#Action 3
Signaldf2['Pass/Fail']=Signaldf2['Pass/Fail'].replace(1,0).astype('int64')
Signaldf2['Pass/Fail']=Signaldf2['Pass/Fail'].replace(-1,1).astype('int64')
3. Data analysis & visualisation:¶
Distribution plot for data with outliers¶
#A. Perform a detailed univariate Analysis with appropriate detailed comments after each analysis.
fig,axes=plt.subplots(18,7,figsize=(20,90))
i=0
for c in Signaldf.columns[:-1]: # finding the axes and plotting distribution
ax = axes[i // 7, i % 7]
sns.distplot(Signaldf[c],bins=5,ax=ax);
i=i+1
Insights¶
Normally distributed Features : Feature 0,1,2,6,14,23,90,115.
Features with median less than mean that is left skew in data : 3,12,15,16,32,33,63,117,132,145,151,155,159,160,161,162,167,182,185,200,201,250,223,426,429,432,438,439,467,476,520,525,572
Feature with median greater than mean that is Right skew in data : 27,570
Features where Majority of data falls in one bin others are outliers: 4,15,16,32,33,63,67,135,142,151,155,167,185,200,201,223,250,429,438,439,467,420,523,550,572,585
Feature 4 - 75th percentile is 1.5 and max is in 1114. hence, there is a outlier.
Feature 24 normally distributed with some dip in the middle
Box plot of data with outliers¶
fig,axes=plt.subplots(18,7,figsize=(20,60))
i=0
for c in Signaldf.columns[:-1]:
ax = axes[i // 7, i % 7]
sns.boxplot(Signaldf[c],ax=ax);
i=i+1
Insights¶
Feature without Outliers: 51,136,418,419,482,486,488,539
Upper Fence outliers : 14,32,43,67,117,122,134,135,137,139,142,150,151,155,159,160,161,162,166,167,182,183,185,188,201,208,218,223,225,250,269,417,423,426,429,432,433,438,439,442,453,460,467,468,472,4476,483,484,485,487,489,491,493,494,496,510,520,523,525,526,527,545,547,548,549,561,572,585
Lower Fence Outliers:27,28,40,133
Both upper fence and lower fence outliers: 0,1,2,3,6,12,15,16,18,21,23,24,33,41,45,6,71,83,88,90,115,129,138,180,200,416,550,562,564,569,570
Belw box plot is plotted after imputing outliers with lower whisker and upper whisker accordingly.
Box plot of data without outliers¶
fig,axes=plt.subplots(18,7,figsize=(20,65))
i=0
for c in Signaldf2.columns[:-1]: #for every categorical calumn , finding the axes and plotting distribution
ax = axes[i // 7, i % 7]
sns.boxplot(Signaldf2[c],ax=ax);
i=i+1
Insights¶
Outliers are handled
Normally distributed Features : Feature 0,1,2,6,14,23,90,115.
Left Skew: 3,12,15,16,32,33,63,117,132,145,151,155,159,160,161,162,167,182,185,200,201,250,223,426,429,432,438,439,467,476,520,525,572
Right Skew: 27,570
#B. Perform bivariate and multivariate analysis with appropriate detailed comments after each analysis.
Bivariate Analysis¶
Target data analysis
labels = ['Pass', 'Fail']
size = Signaldf['Pass/Fail'].value_counts()
colors = ['green', 'red']
explode = [0, 0.1]
#plt.style.use('seaborn-deep')
plt.rcParams['figure.figsize'] = (4, 4)
plt.pie(size, labels =labels, colors = colors, explode = explode, autopct = "%.2f%%", shadow = True)
plt.axis('off')
plt.title('Target: Pass or Fail', fontsize = 20)
plt.legend()
plt.show()
Signaldf['Pass/Fail'].value_counts().plot(kind="bar");
Insights¶
It is observed that target value are imbalanced they have to be balanced with SMOTE
target pass is 93.36% where as Target fail is 6.64% only
plt.figure(figsize=(15,8))
ax=sns.stripplot(y="12", x="18", hue='Pass/Fail' ,data=Signaldf2)
plt. title("12 Vs 18 distinguished by pass/Fail")
plt.show()
Insights¶
feature 12 and 18 is positively correlated and pass and fail are distributed across the data
plt.figure(figsize=(15,8))
sns.swarmplot(x="0", y="1", hue='Pass/Fail' , data=Signaldf2)
plt. title("0 Vs 1 distinguished by pass/Fail")
plt.show()
There seems to no correlation between the two feature which are near[0,1] to each other
Multivariate Analysis¶
sns.pairplot(Signaldf2.iloc[:,0:16]);
Insights (first 15 features)¶
Since multicolinearity data was removed with threshold of 7.5 ; Most of the data are less correlated. there are only few positive corelation observerd in data [ featre 12 and 18 are positively correlated ]
sns.pairplot(Signaldf2.iloc[:,16:31]);
